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Abstract. The management of contemporary software projects is unfeasible without the support 
of a Project Management (PM) tool. In order to enable the adoption of PM tools in practice, teach¬ 
ing its usage is important as part of computer education. Aiming at teaching PM tools, several 
approaches have been proposed, such as the development of educational PM tools. However, such 
approaches are typically limited with respect to content coverage and instructional support. In this 
context, an important technique is the provision of instructional feedback, which is essential in 
order to help the students to learn based on the evaluation of their own actions. In order to take ad¬ 
vantage of this technique, this article proposes its employment in an Instructional Unit, being inte¬ 
grated into the PM tool dotProject+, providing automated feedback based on the project plan being 
developed with the tool. This technique has been evaluated through a series of case studies. 

Keywords: project management; PMBOK; project management tool; dotProject; feedback; 
teaching; learning. 


1. Introduction 

Project Management (PM) is crucial for the success of software projects, being nec¬ 
essary to properly manage the project resources and its constraints in order to meet 
the project requirements (PMI, 2013a). Currently, PM is an extensive discipline, being 
organized in knowledge areas in accordance with the PMBOK guide (PMI, 2013a), 
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which is widely accepted by industry and professionals in this area (Mishra and Mishra, 
2013; Car et al ., 2007). Moreover, considering the complexity of contemporary software 
projects, their management is unfeasible without the support of a software tool (Fabac 
et al, 2010). in this context, a PM tool is a software system that may support the PM 
process. Examples of PM tools include proprietary alternatives, such as MS-Project and 
Oracle Primavera, as well as open-source alternatives, such as project.net and dotProject 
(Pereira et al., 2013). Besides the existence of stand-alone solutions, today web-based 
PM solutions are required in order to facilitate the collaboration and information sharing 
among project stakeholders (Fabac et al., 2010). Yet, most web-based PM tools do not 
support the PM process as defined by PMBOK (Pereira et al, 2013), limiting the assis¬ 
tance to just some part of it (Mishra and Mishra, 2013), such as schedule development, 
risks identification, cost estimation, among others (Car et al., 2007). In this context, 
the open-source PM tool - dotProject has been enhanced to be aligned with PMBOK 
processes, creating dotProject+ as a low-cost and web-based alternative (Gon?alves and 
Wangenheim, 2016a). 

Thus, besides teaching knowledge on PM and interpersonal skills (PMI, 2013a), PM 
education should also cover the usage of PM tools, as considering the need to adopt 
some PM tool for managing software projects in practice, professionals need to know 
how to use them. Consequently, an important competency of project managers and/or 
Software Engineering (SE) professionals is the usage of PM tools (PMI, 2013a; IEEE, 
2014), and the demand for teaching this competency is addressed by the ACM/IEEE 
curriculum guidelines for Computer Science (ACM and IEEE, 2013). 

Currently, for the teaching of PM tools there are two alternatives - using a profes¬ 
sional PM tool, such as MS-Project or dotProject, being developed for the support of 
PM in practice, yet do not provide further educational support (Car et al., 2007). On the 
other hand, some specific PM tools have been developed for teaching, such as ProMES 
and ppcProject (Salas-Morera et al., 2013; Reid and Wilson, 2007). Yet, these tools are 
focused strongly on teaching and are not made for managing real software projects, de¬ 
manding the student to learn some professional PM tool during their professional tasks 
(Gon?alves and Wangenheim, 2015a). 

Furthermore, it is not sufficient to just have a PM tool with educational features, as in 
order to effectively teach this competency a complete Instructional Unit (IU) is required. 
An IU is a set of classes and instructional materials, designed to teach certain learning 
goals in a specific educational context (Hill et al., 2005), including a PM tool as part 
of the IU’s materials. Some IUs have proposed the adoption of educational PM tools 
(Salas-Morera et al, 2013; Reid and Wilson, 2007), which typically offer several exer¬ 
cises to be solved by students, and also automate student answer evaluation. In this way, 
these tools may assist in the teaching of part of the IU’s content, delivering exercises 
to students according to their progress, and also assist with the immediate evaluation of 
student performance. 

However, previous researches (Gon?alves and Wangenheim, 2015b) have identified 
that these existing IUs are rather focused on specific parts of the PM process, address¬ 
ing mostly time and human resource (HR) management. Thus, in order to provide a 
more comprehensive support, improvement opportunities include the coverage of all PM 
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knowledge areas, the enhancement of professional PM tools for education, as well as the 
adoption of more effective instructional strategies (Gagne et al., 1992). 

As an attempt to improve the teaching of PM tools, the IU PiMENTO (Project Man- 
agemENt TOols) (Gonsalves and Wangenheim, 2016a) has been developed, focusing on 
initiating and planning process groups. Among its materials this IU uses an enhanced 
version of one of the most popular open-source tools for PM, dotProject (http: //dot- 
project .net), providing support for the initiating and planning process groups in full 
alignment with PMBOK (PMI, 2013a). Moreover, even though this PM tool is designed 
for professionally manage software projects, it was also enhanced for educational pur¬ 
poses. Also, as part of the IU’s materials the PM initiating and planning process has been 
modelled, being presented in a set of slides for the content explanation in class. The edu¬ 
cational strategy of this IU consists of expositive lectures and practical classes, in which 
students initiate and plan a software capstone project. In these practical classes, students 
utilize a PM tool in computer labs to realize their assignments. 

Furthermore, in order to improve students' learning and motivation it is important 
to also enhance instructional techniques, such as instructional feedback (Kleij et al., 
2012; Kretlow and Bartholomew, 2010). Instructional feedback provides a response 
to students’ actions while doing the practical exercises, indicating how they may cor¬ 
rect or improve them, in order to achieve the IU’s learning goals (Kleij et al, 2012). 
This feedback is normally provided manually by the instructors (Gagne et al, 1992). 
However, giving individual feedback to each student may become impracticable due 
to the demanded effort (Kretlow and Bartholomew, 2010). For this reason, instructors 
are often only able to provide a superficial feedback to students, or some generic feed¬ 
back to the class as a whole (Gagne et al., 1992; Kretlow and Bartholomew, 2010). 
Thus, a practical alternative for providing a customized and individual feedback is 
automatizing its delivery to students with the assistance of a software tool (Chua and 
Balkunje, 2012). 

There are many ways to design an instructional feedback technique, varying the defi¬ 
nition of the proper moment to deliver a message or how to approach the students (Kleij 
et al., 2012; Kretlow and Bartholomew, 2010). For example, for some instructional ac¬ 
tivities immediate feedback may be suitable to assist students when carrying it out; how¬ 
ever, it may also interrupt their attention. Likewise, a feedback approach may explicitly 
highlight students’ mistakes or providing explanations about the content, but leaving the 
students to find their errors by themselves. 

Some of these techniques already have been adopted by previous studies (Chua and 
Balkunje, 2012), proposing feedback messages to be elaborated by instructors, other stu¬ 
dents, or even by a software tool. When the feedback message is elaborated by the soft¬ 
ware itself, it is usually automatic delivered to students (Vivian et al., 2015). However, 
even tools providing an automatic evaluation of the PM tool usage by students, still do 
not fully explore the instructional feedback potential, presenting just simple messages, 
normally highlighting students mistakes, but without providing explanations about how 
to improve or examples that may assist in their understanding (Gregoriou et al, 2010). 
Moreover, none of these approaches have been customized for teaching PM aligned with 
the PMBOK (Gon£alves and Wangenheim, 2016b), lacking the coverage of several PM 
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knowledge areas. Additionally, none of the previous studies includes the introduction of 
some instructional feedback technique in a professional PM tool, focusing rather exclu¬ 
sively on educational tools, diminishing the potential contribution of such techniques to 
the actual students’ demands. 

In this context, this article presents the design of an instructional feedback technique 
integrated into an IU for teaching PM tools, and its implementation as an add-on module 
as part of dotProject+. It provides students with recommendations, examples and expla¬ 
nations about the PM initiation and planning process, based on their interaction with the 
PM tool functionalities. For example, when students are carrying out risk planning, they 
receive feedback messages presenting risk examples, explanations about risk manage¬ 
ment strategies, or feedback messages criticizing their mitigation and contingency plans. 
The contribution of such a technique is evaluated through a series of case studies, apply¬ 
ing the IU in four undergraduate computing courses during 2016. 

Section 2 introduces the main concepts related to this article, and Section 3 presents 
related studies, including existing IUs that employ some instructional feedback tech¬ 
nique for teaching PM tools. Section 4 presents the research methodology adopted in 
this study. The study is defined in Section 5. Section 6 presents the PiMENTO IU and in¬ 
troduces the proposed instructional feedback technique. The evaluation of the proposed 
technique is defined in Section 7 and its execution is presented in Section 8. In Section 9 
the collected data is analyzed and discussed, presenting the conclusions in Section 10. 

2. Background 

2.1. Project Management 

Project Management (PM) organizes project activities and its resources in order to meet 
the project objectives. According to the PMBOK (PMI, 2013a), the PM process is struc¬ 
tured in 5 process groups, from initiation to closure (Fig. 1). 

As a first step to improve the teaching of PM tools, this study targets the first process 
groups: initiating and planning. The initiating process group addresses processes for 


Monitoring & Controlling Processes: 

Monitor and adjust the project 
performance through corrective actions. 


Establish project goals and 
* nA define the 
the project 


Planning Processes: 


Execution Processes: 

Conduct the necessary 
work to carry out the 
project activities. 


Initiating Processes: 
Start a new project 
or phase and obtain 
authorization for its 
execution. 



Closing Processes: 

Finish all project 
activities or phases 
in a formal way. 


Fig. 1. PM process groups (PMI, 2013a). 
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Table 1 

PM knowledge areas (PMI, 2013a) 


Knowledge area Processes to: 


Integration 

Scope 

Time 

Cost 

Quality 

Human Resource 
Communication 

Risk 

Procurement 

Stakeholder 


Identify, define, combine, unify, and coordinate PM processes and PM activities. 
Ensure that the project addresses the entire work and meets all its requirements. 

Plan and control the activities that will be carried out during the project so it concludes 
within the deadline. 

Plan, estimate, and control project costs, so it concludes within the approved budget. 
Define the responsibilities, goals and quality policies so the project meets the needs 
that have initiated it. 

Organize and manage the project team. 

Ensure the generation, collection, distribution, storage and final destination of project 
information. 

Identify, monitor and control the project risks. 

Buy or contract products, services or any resources that are not available as project 
internal resources. 

Identify and manage the stakeholders and their expectations. 


specifying project goals, justifying the need for the project execution, defining its mile¬ 
stones and outcomes, as well as obtaining the commitment of the stakeholders. Its main 
output is the project charter, which has to be approved by the project sponsor, ensuring 
that the necessary resources will be available for the project execution. 

The planning process group covers processes for elaborating the project plan, a docu¬ 
ment defining all necessary information to carry out the project. This project plan should 
not be limited to document the project activities and its resources, but also addresses 
several other topics, such as project costs, risks and procurement demands, covering all 
PM knowledge areas (Table 1), which are essential to properly manage a software proj¬ 
ect (PMI, 2013a; Car et ah, 2007; Salas-Morera et al., 2013). 

The PM process defined by PMBOK includes relevant PM techniques for software 
projects, being among the most adopted approaches in software industry (Eastham et ah, 
2013). Its relevancy for software projects is also highlighted by Project Management 
Institute (PMI), creating an extension of PMBOK for software (PMI, 2013b), detailing 
particularities of PM techniques application in this context. 

Conducting the PM process may be very complex and demand considerable resourc¬ 
es of an organization. To assist in its execution, many PM tools have been developed, 
taking advantage from technology for supporting its execution. 


2.2. Project Management Tools 


Currently, there are a wide variety of PM tools with diverse functionalities and char¬ 
acteristics, such as MS-Project, Primavera, dotProject, Project.net, etc. (Mishra and 
Mishra, 2013). Their functionalities, for instance, may support the whole PM process 
or just some activities related to only one PM knowledge area, for example, registering 
and tracking work hours (Cicibas et ah, 2010). The scope of the provided functionalities 
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may influence in the usage of these tools for teaching, as they may limit the content to 
be addressed. In this context, analyzing existing open-source PM tools the widest (yet 
not complete) support is provided by dotProject (Pereira et al., 2013). Moreover, none 
of these open-source options offers any educational support to assist students and/or 
instructors when utilized in classroom (Mishra and Mishra, 2013; Cicibas et al., 2010). 
However, being open-source, these tools may be customized to meet educational de¬ 
mands. On the other hand, proprietary alternatives, such as Oracle Primavera, provide a 
complete support for the PM process, but, besides requiring monetary investments, they 
also do not include any educational support (Fabac et al., 2010). 

Moreover, in terms of platform, there exist stand-alone tools (mono-user and accessed 
via desktop) or web-based tools (multi-user and accessed via web browser), in practice, a 
web-based PM tool has to be used in order to properly manage a software project, due to 
its support to the collaborative work and the sharing of information (Fabac et al., 2010; 
Pereira et al, 2013; Bhattacharya, 2013). However, the adoption of a web-based PM tool 
requires this tool to be installed on a web server and Internet access during classes. 

Today in practice in the IT industry, composed mainly by Small and Medium En¬ 
terprises (SMEs) (The Standish Group, 2015), with limited resources, open-source and 
web-based PM tools are a feasible alternative (Mishra and Mishra, 2013). Thus, com¬ 
monly PM tools adopted by such organizations are: dotProject, GanttProject, RedMine, 
among others (Mishra and Mishra, 2013; Pereira et al., 2013; Cicibas et al., 2010). 
Hence, the teaching based on such tools is a reasonable choice to better prepare students 
for their professional tasks (Bhattacharya, 2013). 


2.3. Teaching of PM Tools 


Knowing how to use PM tools is a necessary competency for project managers (PMI, 
2013a) and, as described in the Software Engineering Body of Knowledge (SWEBOK) 
(IEEE, 2014), it is also important for SE professionals. The need for its learning is also 
addressed by the ACM/IEEE curriculum guidelines for Computer Science (ACM and 
IEEE, 2013), specifying that students have to learn how to use a PM tool to develop a 
project schedule, to allocate human resources, to perform risk analysis, to monitor the 
project performance, etc. 

Typically, IUs for teaching PM tools include activities in which students have to 
solve problems applying some PM technique with a tool (Gonpalves and Wangenheim, 
2015b). Among these techniques typically are (PMI, 2013a; Car et al, 2007; Reid and 
Wilson, 2007): 

• Critical Path Method (CPM): to identify project activities that cannot be delayed 
without affecting the deadline. 

• Program Evaluation and Review Technique (PERT): to calculate the estimated 
effort to carry out an activity based on three estimates (worst case, most common 
case, and best case). 

• Earned Value Analysis (EVA): to measure project performance and progress in an 
objective manner. 
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Table 2 

Addressed PM knowledge areas and process groups by the PM techniques typically taught (PMI, 2013a) 



Integra- Scope 
tion 

Time 

Cost 

Qua¬ 

lity 

HR 

Commu¬ 

nication 

Risk Procure- Stake- 
ment holder 

Initiating 

- 

- 

- 

- 

- 

- 

- 

Planning 


CPM, 

PERT, 

RL 



RACE 

RL 



Executing 

- 

- 

- 

- 

- 

- 

- - 

Monitoring & 
Controlling 

EVA 

EVA 

EVA 

— 


— 

_ _ — 

Closing 

- 

- 

- 

- 

- 

- 

- - 


• Resource Leveling (RL): to adjust start and finish dates based on resource con¬ 
straints, with the goal of balancing resource demand and availability. 

• ACI Matrix: describes the participation by various agents and their roles in com¬ 
pleting project activities. 

These activities typically involve the use of some professional PM tool (Car et al, 
2007), or even some educational PM tool (Salas-Morera et al., 2013; Reid and Wilson, 
2007). However, the teaching of PM tools should not be limited to the execution of spe¬ 
cific PM techniques (Table 2), but addresses its general usage, covering the PM process 
as defined by PMBOK. 

Thus, the teaching of PM tools should address the usage of functionalities to sup¬ 
port all process groups and knowledge areas, enabling the student to learn cognitive 
competencies up to the application level following Bloom’s taxonomy (Bloom, 1956) 
in order to be able to use a PM tool effectively. In this respect, several IUs rather seem 
to take students only up to the remembering and comprehensive level (Salas-Morera 
et al., 2013; Reid and Wilson, 2007; Gonsalves and Wangenheim, 2015b), due to their 
instructional strategy basically relying on theoretical lectures, not involving any hands- 
on activities using a PM tool. 

In addition, beyond the practical use of PM tools in the classroom, its teaching can 
be improved by the employment of instructional feedback as an important technique for 
student learning (JangHyeon et al., 2002). 


2.4. Instructional Feedback 

Learning is the result of the human capability to acquire, transform and evaluate infor¬ 
mation based on the experience, causing a permanent change in his behavior (Bruner, 
1977). Learning may occur at any time, even during daily situations, influenced by ran¬ 
dom events. However, for a controlled learning, teaching is introduced, providing means 
for the systematic development of competences (Bruner, 1977; Branch, 2009). This 
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1. Gaining attention 

2. Informing the learner of the objective 

3. Stimulating recall of prior knowledge 

4. Presenting information 

5. Providing guidance 



8. Assessing performance 

9. Enhancing retention and transfer 


Fig. 2. Gagne events of learning (Gagne et al., 1992). 


teaching/learning process can be divided in events as proposed by Gagne et al. (1992) 
(Fig- 2). 

An important event in the learning process is the provision of feedback for stimulat¬ 
ing student reflection about their own actions (Schilling, 2015). It provides information 
to students, criticizing their way of acting and thinking, consequently promoting their 
learning (JangHyeon et al, 2002). This technique promotes learning through errors, 
using students’ mistakes to concentrate the provided explanations exactly on topics 
of the IU content, towards which students present learning problems (Schilling, 2015; 
Bilal et al., 2012). 

There are different techniques to promote feedback, in accordance to the adopted 
instructional strategy (Branch, 2009), defining the feedback goal (assist students to 
improve their responses or marking it), how to approach students (providing extra ex¬ 
planations or highlighting their mistakes), and also the moment to deliver the message 
to students. 

Feedback may have different goals, being formative or summative (Segers et al., 
2008). Formative feedback is designed in order to instruct students during an activity, 
assisting in its improvement before the delivery. On the other hand, summative feedback 
is given at the end of an activity, and is normally used for grading (Segers et al., 2008; 
Martens et al., 2010). 

in relation to how students will be approached by a feedback message, it may be 
provided by verification, informing whether the student response is correct or not, or by 
elaboration, providing information assisting in the improvement of student response, 
but not necessarily judging its correctness (Segers et al., 2008). A feedback technique 
may also focus on different aspects of student response correctness, as it may be posi¬ 
tive (highlight the students correct responses), negative (highlight the students mis¬ 
takes), or constructive (not only informing if a certain response is wrong or incomplete, 
but providing means for the student to improve it) (Kleij et al., 2012; Kretlow and 
Bartholomew, 2010). 
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Feedback may be delivered immediately after some student action, or delayed, some¬ 
time after the conclusion of an activity (Scheeler et al., 2010). Yet, the moment of de¬ 
livering feedback is typically related to the adopted communication channel, which may 
vary from computer-based to oral or written by the instructor (Bilal et al, 2012; Martens 
et al, 2010). There are different manners in which an instructional feedback can be 
delivered to students. An instructor can provide individual feedback to each student, or 
provide a general feedback to the class as a whole. 

Specifically when teaching PM tools, immediate and automated feedback has been 
an interesting alternative, due to the constant demand of interaction between students 
and the tool’s functionalities (JangFlyeon et al., 2002; Schilling, 2015; Ojiako et al, 
2011). In order to know if such functionalities are being completely (i.e. addressing all 
PM knowledge areas) and correctly utilized (including the correct information), instruc¬ 
tional feedback is an important technique for learning (Gagne et al., 1992; Gregoriou 
et al., 2010). In this context, students can receive feedback messages during their inter¬ 
actions with the PM tool, aiding them to execute the PM process, and avoiding stagna¬ 
tion in specific parts of the process. Being automated, it is expected that students receive 
the feedback messages at the right moment, addressing the content properly. A further 
advantage is that in such feedback does not depend on the instructor availability, neither 
his/her constant analysis of students’ behaviour. 


3. Related Studies 

In order to identify existing IUs for the teaching of PM tools involving some instruc¬ 
tional feedback technique, we conducted a Systematic Literature Review (SLR) (Gon- 
9 alves and Wangenheim, 2016b). We searched for studies published in academic jour¬ 
nals and conference proceedings in relevant digital libraries. Yet, only few relevant 
studies were found, indicating that, despite the existence of a reasonable amount of 
studies about the teaching of PM tools, only very few report the adoption of some in¬ 
structional feedback technique. 

Gregoriou et al. (2010) introduce an interesting feedback technique, which is includ¬ 
ed in an IU for teaching PM techniques, such as CPM, PERT, and RACI matrix, focusing 
on exercises carried out in the educational PM tool - ProMES. The adopted instructional 
feedback technique delivers feedback messages immediately after each submitted re¬ 
sponse. These messages are focused on highlighting the students’ mistakes, following 
the negative feedback approach. As a result, students became aware of their errors, and 
tried again to find another solution. Students indicated that the feedback technique was 
helpful, along with the feature that highlights the existing problems without revealing 
the right answer. It was highly appreciated, since they had the opportunity to try again 
until they produced the correct result. Yet, the impact of the feedback technique remains 
vague as only an ad-hoc evaluation has been reported based on verbal comments of the 
students on its contribution to their learning. 

The instructional feedback techniques reported by other studies may be classified 
into two categories. One type of techniques evaluates students’ decisions when running 
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simulations of the execution of a project plan (Chua and Balkunje, 2012; Gregoriou 
et al., 2010; Rodriguez et al., 2006). In these studies, feedback is automatically delivered 
after each simulation, reporting issues with regard to the trade-off between project time, 
cost, scope, and quality. Additionally, they adopt the elaboration feedback approach, 
suggesting possible improvements that the students could make before a new simula¬ 
tion. Likewise, the verifying approach is adopted, indicating whether students’ actions 
are correct or not. This kind of feedback is normally presented by visualizing progress 
tracking, based on EVA indicators (Souza et al., 2015) or remaining time and budget. In 
the reported studies these feedback techniques were typically evaluated by case studies, 
collecting data via questionnaires after the lectures. The findings of these studies high¬ 
lighted the positive impact of the instructional feedback, especially in relation to student 
motivation. 

A second category of studies focuses on instructional feedback provided through 
communication channels of collaborative PM tools (Vivian et al., 2015; Tachikawa 
et al., 2013; Xiao et al., 2008), such as wiki, forums, chats, or shared whiteboards. In 
this case, the instructional activities demand the usage of these communication channels 
to promote discussions while managing a software project. PM decisions and rationales 
are discussed and registered by some students, while others analyse and comment them. 
Consequently, the feedback is elaborated by students themselves, promoting collabora¬ 
tive learning (Caballe et al, 2010), where the instructor acts as a mediator, evaluating 
the promoted feedback and interfering when students are not able to evaluate, wrongly 
criticize, etc. This technique, thus, demands considerable effort from instructors in order 
to monitor students’ evaluations. These studies also were evaluated through a case study, 
collecting data through system logs. Findings demonstrate that the students may evalu¬ 
ate the effectiveness of actions of their peers, through the discussion of the results and 
even find alternatives for improvement. 

The processes addressed by these lUs are rather focused on time and HRs manage¬ 
ment. IUs either request the students to elaborate project plans or to analyse existing 
ones. Additionally, the communication knowledge area has also been superficially ad¬ 
dressed by some studies, due to the adoption of collaborative PM delivering project 
artefacts to specific stakeholders, namely customers, sponsor or team members, repre¬ 
sented by other students, instructors, or even chatterbots. However, this communication 
is usually focused on disseminating information during the project execution, but not 
elaborating a project communication plan. 

In summary, we can observe that the existing IUs for teaching PM tools do not com¬ 
pletely cover all PM knowledge areas in conformance with the PMBOK. Moreover, 
analysing the tools adopted by these studies, it has been evident that so far the employ¬ 
ment of some instructional feedback technique demands the adoption of an educational 
PM tool. However, such tools have demonstrated to be focused on specific parts of the 
PM process, limiting the contribution of the instructional feedback in relation to other 
parts of the PM process, as demanded by a professional use of PM tools. Thus, in order 
to optimize learning in a more comprehensive and relevant way, it may be advantageous 
to employ some instructional feedback technique in a professional PM tool covering all 
PM knowledge areas in compliance with PMBOK. 
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4. Research Design 

This study aims at the development and evaluation of an instructional feedback tech¬ 
nique integrated in an IU for teaching a PM tool. Therefore, we conducted an explor¬ 
atory case study (Yin, 2014) (Fig. 3) in order to understand the students’ perceptions 
about the contribution of the instructional feedback during the 1U. 

The case study is performed according to the procedure proposed by Yin (2014) and 
Wohlineta/. (2012): 

• Study definition. The study is defined, including the establishment of its goal, and 
defining how the goal is going to be evaluated in terms of analysis questions and 
metrics, following the GoaPQuestion/Metric (GQM) approach (Basili etal., 1994). 
Based on the measurement plan, data collection instruments are developed. 

• Study execution. The study execution follows the ADDIE approach (Branch, 
2009), addressing the design, development and application of the instructional 
feedback technique in an 1U. This includes the customization of an IU for teach¬ 
ing a PM tool, in which the instructional feedback technique is employed. Flere, 
we enhance PiMENTO (Gonsalves and Wangenheim, 2016a), an IU focused on 
the teaching of a PM tool aligned with PMBOK processes, adopting the open- 
source PM tool dotProject+, which is evolved to include the instructional feed¬ 
back technique. The IU is applied in practice in PM courses collecting data as 
defined by the measurement plan. 

• Analysis and Interpretation. The study is analysed through descriptive statis¬ 
tics methods, evaluating the collected data regarding student perception about the 
quality of the instructional feedback technique. 

This study was approved by the ethics committee CEPSH/UFSC - under the n°. 
47734215.9.0000.0121. 
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Fig. 3. Overview on research methodology. 
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5. Study Definition 

The objective of this study is to analyse the quality of the instructional feedback tech¬ 
nique employed in PiMENTO IU with respect to content, strategy, learning, materials, 
and user experience from the point of view of the students in the context of higher edu¬ 
cation computing courses. In order to achieve this objective, we carried out an explor¬ 
atory case study, applying the IU in higher education computing courses, analysing the 
students’ perception about the instructional feedback technique. 

Following The GQM approach, analysis questions were defined with respect to each 
quality aspect: content (the topics addressed), strategy (the moment the messages are de¬ 
livered to students and how they were approached by the messages), learning (students’ 
perception of learning through the feedback messages), materials (quality of the feed¬ 
back technique implemented into dotProject+), and user experience (how students ap- 


Table 3 
GQM plan 


Goal: Analyse the instructional feedback technique in the PiMENTO IU, aiming at evaluating its quality in 
relation to content, strategy, learning, material and user experience, under students’ point of view, in the context 
of higher education computer courses. 

Metric 

Questioner Item 

AQ01: How appropriate is the content presented by the instructional feedback technique? 

Ml. Median of students’ perception regarding the 
content coverage. 

M2. Median of students’ perception regarding the 
content depth. 

The feedback messages delivered by dotProject+ provide 
information for all knowledge areas (coverage). 

The feedback messages delivered by dotProject+ expose 
the content clearly and with enough information for my 
understanding (depth). 

AQ02: What is the quality of the instructional strategy? 

M3. Median of students’ perception regarding the 
time for feedback delivery. 

M4. Median of students’ perception regarding how 
feedback messages approach them. 

I consider the moment that the feedback messages are 
delivered to be appropriate. 

I consider the format of the messages and the employed 
terminology is appropriate. 

AQ03: What is the contribution of the instructional feedback technique for student learning? 

M5. Median of students’ perception regarding the 
instructional feedback technique contribution in 
their learning. 

The explanations presented by the feedback messages 
assisted in the improvement of my learning. 

AQ04: What is the quality of the instructional feedback add-on module implemented in dotProject+? 

M6. Median of students’ perception regarding the 
quality of feedback technique as an instructional 
material. 

I consider the feedback messages delivered during 
the usage of dotProject+ as a useful instrument for my 
learning. 

AQ05: What is the user experience promoted by the instructional feedback technique? 

M7. Median of the recommendation level of the 
instructional feedback technique by students. 

M8. Median of students’ perception of motivation 
during the IU activities owing to the instructional 
feedback mechanism. 

I would recommend the usage of the instructional 
feedback technique to other students that need to learn 
PM tools. 

The feedback messages delivered by dotProject+ have 
increased my motivation to improve the project charter 
and the project plan. 
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preciated to interact with this technique and its effects on students’ motivation). A com¬ 
plete list of analysis questions and metrics is presented in Table 3. 

Data is collected through a questionnaire composed of items as presented in Table 3. 
The questionnaire items are defined as affirmations with responses on a 5-point Lik¬ 
ert scale (Wohlin et al., 2012), ranging from -2 (totally disagree) to 2 (totally agree). 
The questionnaire also contains open questions asking students to indicate strengths, 
improvement opportunities and other comments regarding the instructional feedback 
technique. Additional data has been collected by analysing the students’ works as well as 
through log files of their usage of the tool dotProject+. This data is used in order to anal¬ 
yse to which degree the students were able use a PM tool to develop the project charter 
and the project plan, as well as, to which degree the instructional feedback technique has 
been utilized by students. Data collection has been carried out just after the application 
of all IU lectures. 


6. PiMENTO IU 


The PiMENTO (Project ManagemENt TOols) IU aims at teaching a PM tool during 
project initiation and planning in conformance with the PMBOK, covering all PM 
knowledge areas. 


6.1. Context Analysis 


The IU is designed to be used as part of PM or SE courses. In this context, the target 
audience is typically composed of students between 18 and 30 years, who do not have 
prior experience with PM tools but who are often studying or already have studied some 
SE course. 

The teaching of PM tools is usually done as part of a SE course or in PM specific 
courses. These courses, besides teaching the usage of PM tools, also cover the teach¬ 
ing of PM concepts, including knowledge, skills, and techniques to carry out the PM 
process. In this context, PM tools are introduced as an important instrument to assist the 
PM process execution. 

These higher education classes are typically conducted in classrooms and/or com¬ 
puter labs with Internet access. 


6.2. Design of the Instructional Unit 


Based on the ACM/IEEE curriculum guidelines (ACM and IEEE, 2013), the learning 
objectives of the PiMENTO IU are defined focusing on teaching the usage of a PM tool 
during project initiating and planning: 

a) After the students completed the IU, they should know how to use a PM tool to 
elaborate a project charter for a software project. 
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b) After the students completed the IU, they should know how to use a PM tool to 
elaborate a software project plan, addressing all PM knowledge areas in confor¬ 
mance with PMBOK. 

The IU content is designed to address the PM tool’s functionalities to support the 
initiating and planning process groups as well as theoretical knowledge with respect to 
these processes. 

The instructional strategies adopted include expositive classes and experiential 
learning (Branch, 2009; Dick and Carey, 2006). The unit is sequenced by alternating 
the presentation of theoretical knowledge thorugh expositive lectures, and practical 
classes, in which students use a PM tool to carry out the respective part of the PM 
process presented the previous theoretical lecture. The practical classes begin with 
an expositive presentation, in which the instructor explains how to use the PM tool 
functionalities to support the respective part of the PM process, and then, students use 
the PM tool in order to elaborate their own project charters and project plans. In total, 
the IU is composed of 13 lectures with duration of l:40h each. The planned content, 
instructional activities and materials designed for each lecture are presented in the IU 
syllabus (Table 4). 


Table 4 
IU syllabus 


Class Content Instructional method Resources and materials 
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Class 

Content 

Instructional method 

Resources and materials 

5T 

• HR planning. 

Expositive lecture. 

• 

Proj ector/N otebook 


• Communication planning 

Exercises: related to HR, 

• 

Slides 


• Quality planning. 

communication, quality, and 




• Procurement planning. 

procurement planning. 



5P 

• HR planning. 

Expositive lecture: explanation 

• 

Computers with internet 


• Communication planning 

about tool functionalities. 

• 

Projector 


• Quality planning. 

Experiential learning: dotProject+ 

• 

dotProject+ 


• Acquisition planning. 

usage by students. 

• 

Usage guide slides 

6T 

• Risk planning 

Expositive lecture. 

• 

Proj ector/N otebook 


• Stakeholders planning. 

Exercises: related to stakeholders 
planning. 

• 

Slides 

6P 

• Risk planning 

Expositive lecture: explanation 

• 

Computers with internet 


• Stakeholders planning. 

about tool functionalities. 

• 

Projector 


• Project plan exportation. 

Experiential learning: dotProject+ 

• 

dotProject+ 



usage by students. 

• 

Usage guide slides 

- 

- 

Project charter and plan deliver. 

• 

Computer with internet 

7 

• Project charter 

Presentation: students present their 

• 

Projector/Notebook 


• Project plan 

work to other class members. 

• 

Slides 

- 

- 

Project charter and plan correction. 

• 

Computer with internet 


The main activity that the students carry out is the elaboration of a project charter 
and project plan using dotProject+. The theme for this project is a capstone project, 
which every student must develop when concluding their undergraduate computing 
program. During the elaboration of the project charter and the project plan, students 
are organized in groups composed of up to 4 members each. Its grouping enables them 
to discuss, for example, effort estimations to carry out project activities and possible 
risks. 

The students’ learning is assessed by evaluating the students’ project charters and 
plans (written and oral presentation) using a rubric and exam questions. During the 
oral presentation of the project charter and plan, students also receive feedback from 
the instructor, indicating strengths and weaknesses. They can use these suggestions to 
improve the orally presented project charter and plan before submitting a final written 
version, which is used for grading. 


6.3. Development of the Instructional Material 


Based on the design of the PiMENTO 1U, instructional materials have been developed, 
including the enhancement of a PM tool and a usage guide presenting the use of its func¬ 
tionalities. In the context of this IU, the PM tool - dotProject+ was enhanced, including 
the set of functionalities to fulfil the IU demands. 
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Fig. 4. Overview on dotProject+. 


6.3.1. dotProject+ 

As one of the most popular open-source web-based PM tools, we have adopted dotProj¬ 
ect for teaching. Here, we are specifically using an extension, dotProject+ (Gongalves 
and Wangenheim, 2016a) that provides support to an extensive part of the PM process 
in alignment with PMBOK (Fig. 4). In relation with the initiation and planning process 
dotProject+ provides support for all PM knowledge areas. 

Its main functionalities support the elaboration of the project charter and the elabora¬ 
tion of project plans in alignment with PMBOK. Moreover, dotProject+ generates the proj¬ 
ect plan, being composed by the plans produced for each knowledge area, as a single and 
structured plan. It is also supports the generation of a PDF document of these artefacts. 

6.3.2. Usage Guide 

The usage guide is composed of a set of slides that may be adopted by instructors in 
expositive lectures as well as the students as a reference material during the instructional 
activities. This usage guide is structured trough the PM process represented using the 
BPMN notation (Weske, 2012) (Fig. 5). 

The usage guide presents the usage of the tool step by step demonstrating for each 
activity screens and explaining how to use dotProject+ functionalities to support that 
process activity. 

6.3.3. Instructional Feedback Technique 

Aiming at improving the teaching of PM tools, we propose a new instructional feedback 
technique based on instructional feedback guidelines (Thurlings et al., 2013), experienc¬ 
es reported by related studies (Gongalves and Wangenheim, 2016b), as well as our teach¬ 
ing experiences observing typical students’ errors made during the learning process. 
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dotProject+ 


Educational open-source software for 
Project Management 




Af+er configure HR it is poss b\e lo 
alloca+e +Hem on project ac+rvr+ies 




All activities wh*ch hove rote? estmated it is 
possible to allocate a human resource 


1.1.1 P1md*(fww 



dotProject+ suggest which human resources may 
be allocated, ndcattrig whether they have the 
competency to the estimated rote. 


Fig. 5. dotProject+ usage guide. 


In order to integrate the instructional feedback into the IU and aiming at an optimiza¬ 
tion of the learning process, we decided to design an instructional feedback technique 
that would provide instructions to students based on the project charter and project plan 
being developed with the PM tool. 

The feedback messages are presented immediately to students, while interacting with 
the PM tool, indicating possible omissions and suggestions on what is necessary to com¬ 
plete the project plan. However, in order to keep interruptions minimal intrusive, the 
messages can be read by them when they want. Taking into consideration students’ learn¬ 
ing preferences, it is also possible to turn off the instructional feedback functionality, as 
maybe experienced students do not want to receive feedback during the tool usage. 

We adopt an elaboration approach, focusing on the constructive evaluation of student 
actions. Analysing the project charters and project plans being developed, typical issues 
are identified, e.g. project activities with duration estimated before estimating their ef¬ 
fort; project costs baselines not including a contingency reserve for project risks; risks 
mitigation and contingency actions not matching planned management strategies; etc. 

The feedback messages are categorized according to the respective PM knowledge 
area, as this information may indicate to which knowledge area the issue that needs to 
be corrected is related. The feedback messages are delivered in two formats (Thurlings 
et al., 2013; Fund, 2010): first a short message (presented as a question) is visualized in 
a feedback panel to instigate the student’s curiosity, and the second is presented when 
requested by the student providing a detailed feedback message. 
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The instructional feedback technique follows the workflow as presented in Fig. 6, 
beginning with the students’ interaction with the PM tool, until receiving of the complete 
feedback message. 



tool usage 


Fig. 6. Instructional feedback workflow. 


Table 5 

Examples of general instructional feedback 


PM know- Trigger Feedback short Feedback complete messages 

ledge area message (question) 


Integration When student complete the 
project charter elaboration, 
but it still not authorized. 


Did you remember You have concluded the project charter, 
to obtain the project However, it still needs to be authorized by 
charter approval? the project sponsor to ensure the estimated 
resources will be authorized. You may inform 
the project charter approval using the workflow 
actions available in the project charter form. 


Scope 


When student initiates the Did you already 
activities estimates (effort, estimate the size/ 
duration, and resources) but complexity for all 
still not estimate the work work packages? 
packages size/complexity. 


Observe that still exist work packages which 
size/complexity was not estimated. It is 
necessary before carry out the time and cost 
estimates. 


Time 


When identified that student Do you know that 
create circular dependencies there are circular 
among project activities dependencies in 
(when two activities depend your project? 
on the conclusion of each 
other to start). 


Attention to not create circular dependencies. 
It is when two activities depend on the 
conclusion of each to start. You may divide 
some of these activities for enabling the linear 
sequencing of these activities. 


Cost 


Risk 


After the cost baseline 
creation, and when there 
are risks registered for the 
project, but any risk set in 
the contingency reserve. 


Do you know that 
the financial impact 
of project risks 
may be included 
in the contingency 
reserve? 


During the cost planning is important to 
analyse the contingency reserve. You may 
verity among the project risks, which ones 
have to be included in the contingency reserve, 
and flag them checking the option “Include in 
the contingency reserve”. 


When a prevention plan There are prevention 
has been registered for an plans for eliminated 
“Eliminated” risk. risks? 


Remember that eliminated risks do not exist 
anymore and consequentially do not demand 
to be prevented. 
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Considering that it is necessary to give general feedback (about the PM process) as 
well as specific feedback (about the content of what is being elaborated for the specific 
project), the feedback messages are also categorized in terms of their character (generic 
or specific). The generic messages may be applied on any software project, as they are 
related to the PM process in general, evaluating deviations in the student usage of the 
PM tool and the steps defined for such process. Some examples of general feedback are 
presented in Table 5, indicating the respective knowledge area, the trigger for delivering 
this feedback, and the short and detailed messages. 

On the other hand, specific messages are provided for projects on the specific 1U 
project theme, a capstone project. These messages typically regard the content of what 
is being elaborated, providing suggestions and examples of content. Examples of these 
messages are presented in Table 6. 

The feedback messages are delivered to students when their trigger is fired. The 
status of the triggers are computed based on the evaluation of the content of the stu- 


Table 6 

Examples of specific instructional feedback 


PM knowledge Trigger Feedback short Feedback complete messages 

area message (question) 


Communication After student have There is some perio- During the capstone project it is 
registered two dical meeting with necessary periodical meetings with 

communication frequen- your advisor? the advisor. You may register these 

cies in the project plan, meetings in the communication plan and 

and there is no “monthly” its periodicity, 

or “weekly” frequency. 


Quality 


When student initiated the Have you identified 
elaboration of the quality the rules and norms 
plan, but the field “Norms, to produce a capstone 
policies, and quality project yet? 
guidelines” has less than 
150 characters. 


A capstone project may follow certain 
technical restrictions, as formatting rules, 
structure, and even about its deadline. 
These guidelines are normally defined 
by the educational institution. You may 
register such information in the “Norms, 
policies, and quality guidelines” field. 


HR 


After registering 3 roles in 
the organizational diagram, 
and there is no role for 
advisor, student, or board 
member. 


Were the roles of A capstone project must include roles 
a capstone project for advisor, student, and board members, 
defined? Such roles may be included in the 

organizational diagram. 


Procurement 


Stakeholder 


If the student has less than 
3 items in the procurement 
plan when concluding the 
project plan. 


There are just few 
planned acquisitions. 
Do you already have 
registered the main 
acquisitions for your 
capstone project? 


Remember that capstone projects 
may include acquisition such as 
books, specific software or hardware, 
printing supplies, beyond other items 
to for carrying out experiments or case 
studies. 


If the student has registered Have you already Typically capstone projects include 

less than 3 stakeholders identified the at least the following stakeholders: 

when concluding the stakeholders for the student, advisor, and board members, 

project plan. capstone project? Other stakeholders may vary according 

to the project application area. 
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Mew communication event , New communication channel I New communication freauency 


Fig. 7. Instructional feedback in dotProject+. 


dent’s project charter and project plan. For instance, when a student starts to register 
risks, feedback messages with suggestions of risks examples for capstone projects are 
delivered. 

Implementation of the feedback technique into dotProject+ 

The instructional feedback technique was implemented as a new add-on module for 
dotProject+. In Fig. 7 the steps designed for the instructional feedback workflow (Fig. 6) 
are highlighted, being represented by their respective numbers. 

Fig. 7 demonstrates the functionalities of this add-on module. First, the students inter¬ 
act with dotProject+ functionalities (step 1) and the tool evaluates these actions (step 2) 
firing the triggers. When feedback is delivered, a notification is shown in the bell icon 
(step 3), indicating that there are feedback messages (as well as how many messages are 
given). Once the students click to check the messages, the short messages (questions) for 
the feedback messages are displayed (step 4). In case the student clicks on a short mes¬ 
sage the complete message is displayed in the feedback message area (step 5). 

This add-on module is published as part of dotProject+, under the license GPL v3, 
and is freely available for download in dotmods (https: //sourcef orge. net/proj- 
ects/dotmods/) the official repository for dotProject add-on modules. 


7. Application of the IU 

The created instructional feedback technique has been applied in a series of case studies 
executed during the first and second semester in 2016. It has been applied in 4 different 
classes from 3 educational institutions, including a total of 64 students (Table 7). 
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Table 7 

IU applications using the instructional feedback technique 


Semester 

Educational 

institution 

Program 

Course 

Instructor* 

n° of 
students 

Collected 

answers 

2016-1 

UFSC 

Computer 

Science 

Planning and management 
of software projects 

Instructor A 

14 

10 

2016-2 

UFSC 

Computer 

Science 

Planning and management 
of software projects 

Instructor A 

34 

29 

2016-2 

UFMS 

Computer 

Science 

Project management 

Instructor B 

6 

5 

2016-2 

UFPB 

Computer 

Science 

Software project 
management 

Instructor C 

10 

7 


Legend: 

UFSC - Federal University of Santa Catarina 
UFMS - Federal University of Mato Grosso do Sul 
UFPB - Federal University of Paraiba 
* Due to privacy reasons instructor names have been omitted. 


Data collected in the case studies were pooled in a single sample, and used cumula¬ 
tively in order to evaluated the instructional feedback technique. The pooling of data was 
possible due to the similarity of the courses and the adoption of the same data collection 
instruments. Despite the case studies being carried out in different courses, they are 
compatible, demanding the same prerequisites (e.g. SE course) and containing related 
syllabus, both including the teaching of PM tools aligned with PMBOK. The similarity 
and standardization in terms of definitions, methods, and measurements are essential 
aspects for the pooling of data (Kish, 1994). In this respect, the studies are similar in 
terms of definition (with the objective to evaluate the instructional feedback technique), 
research design (case studies), and context (higher education computer courses). In ad¬ 
dition, the case studies are standardized in terms of measures (quality factors), data col¬ 
lection method (Table 3), and response format (5-point Likert scale). 

During the practical classes, student reaction was observed when interacting with 
the functionalities of the instructional feedback add-on module. These functionalities 
have been explained to students in the beginning of the practical lectures, preceding the 
presentation of dotProject+ support to the initiating and planning process groups. How¬ 
ever, it was observed that in the initial classes, the instructional feedback was not much 
used by the student, as they did not perceive its existence. At this point, the instructors 
stressed again the advantage of the feedback and a wider usage has been observed based 
on dotProject+ log files and students’ feedback. 

After the IU lectures, the students answered the evaluation questionnaire. From a 
total of 64 students, 51 answered the questionnaire, representing a participation rate of 
80%. As their participation was non-mandatory and anonymous, the main reason for 
students not answering the questionnaire was their absence in the respective class. 
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8. Evaluation of the IU 

The quality of the instructional feedback technique is analysed and discussed based on 
the defined analysis questions. 

AQ1. How appropriate is the content presented by the instructional feedback technique? 

About 60% of the students considered the content depth adequately addressed, in¬ 
dicating that the messages are detailed enough for understanding. Yet, some students 
indicate they would prefer to receive more detailed explanations about some topics. 
However, since most students have agreed that the addressed depth is suitable for their 
learning, and at the same time do not overload them with an excess of information, a 
possible improvement could be providing some external references (e.g. IU slides or 
other online materials). Thus, satisfying also students that demand more information for 
their understanding. 

On the other hand, in relation to the content coverage, 60% of the students indicate 
it was not satisfactory, as they did not perceive messages with respect to all PM knowl¬ 
edge areas. Analysing the descriptive answers, it is evident that this could be due to 
problems to visualize the messages as soon as they are available. As presented in Fig. 6, 
the notification of these messages is just a flag in the bell icon. Thus, even with feedback 
messages being delivered for all PM knowledge areas, students may not have noticed 
the existence of such messages. Consequently, several improvements were suggested by 
students in relation to the intrusiveness of the feedback messages, which are discussed 
when analysing the user experience dimension (AQ05). 

AQ02: What is the quality of the instructional strategy? 

The instructional strategy was analysed based on the moment that the instructional 
feedback messages are delivered, and how these messages reach the students. In relation 
to the moment of delivery, the proposed technique has adopted an immediate but not 
intrusive approach. In this regard, about 60% of the students consider it was appropriate 
for their learning. Nevertheless, the other 40% disagrees. When analysing the descrip¬ 
tive answers, it becomes evident that the problem may not be related to the moment of 
delivery, but again with the low intrusiveness of their notification. Specifically, some 
students indicated that they did not perceive the existence of new messages as soon as 
they became available due to the discreteness of the notification. Often they only noticed 
the messages later on, when they were working already on a different part of the PM 
process, which may have complicated for them to return to the issue pointed out in the 
message. Thus, an improvement suggestion is that the feedback message also contains a 
link to the screen to which it refers. 

However, most students agree that the moment of delivery is satisfactory. Once they 
understood how to check for new messages, they were able to read them as soon as they 
became available. They also indicated that the immediate delivery approach was appre¬ 
ciated, as it assisted them in verifying if artefacts they were planning were complete or 
if they had forgotten to execute some step of the PM process. 

The instructional feedback technique has also defined strategies related to how to 
approach the students. The designed approach has been positively evaluated by 66% of 


An Instructional Feedback Technique for Teaching Project Management Tools ... 219 

students. They agreed that the messages questioning them about a certain topic, and later 
presenting detailed explanations and examples, assisted their learning. 

Yet, regarding the instructional strategy, few students informed they would like to 
have more time for practical lectures, thus, obtaining more time to read the feedback mes¬ 
sages. A possible reason for these comments is that many students may have started the 
usage of the instructional feedback technique in the last practical classes, consequently 
having a reduced time to completely explore the feedback messages. Additionally, due 
to time restrictions typically pertaining this kind of IU (Gonsalves and Wangenheim, 
2016a), increasing the quantity of practical lectures may be unfeasible. As an alterna¬ 
tive for students, who demand more time for reading the feedback messages, maybe to 
enable the access of dotProject+ externally to the educational institution, being able to 
continue their studying at home. 

AQ03: What was the contribution of the instructional feedback technique for student 
learning? 

About 65% of the students agreed that it is a useful instrument for their learning, 
its main contribution is the assistance with the comprehension of concepts, providing 
contextualized explanations that reinforce the understanding of PM concepts. Another 
important feature for student learning are the examples provided by the feedback mes¬ 
sages, explaining what is expected for each part of the project charter and project plan. In 
this regard, students appreciated receiving messages dedicated to their capstone project, 
the IU project theme. This specific feedback provides examples, for instance, demon¬ 
strating possible stakeholders, activities, risks, and acquisitions typically required for 
this kind of project. Hence, these examples instigate students to elaborate their project 
plan and also to discuss other alternatives in order to improve the completeness of the 
PM artefacts. 

Yet, although about 30% of the students disagree that such technique contributes to 
their learning, a possible cause could be it not being identified. This may be due to the dif¬ 
ferent learning preferences, and/or related to the discreteness of the feedback messages. 

AQ04: What is the quality of the instructional feedback add-on module implemented in 
dotProject+? 

When asked about the quality of the instructional feedback add-on module as a di¬ 
dactic material, 75% of the students considered it an important instrument for their learn¬ 
ing. Some students indicated that, when they understood how to check the notification 
of new feedback messages, they appreciated the constant update with new messages as 
they interacted with dotProject+. They also emphasized the completeness of dotProject+ 
in relation to the set of functionalities provided, enabling them to develop the complete 
project plan with only one tool. However, as the constant updates with feedback mes¬ 
sages takes a certain time, some students expressed the need to have more time to check 
the feedback messages properly. 

AQ05: What was the user experience promoted by the instructional feedback technique? 

The user experience was analysed based on the recommendation the students would 
make to other students with similar educational needs. Nearly 80% of the students agreed 
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that they would recommend it to their peers, as they considered it a positive experience. 
They indicated that it assisted to detect errors and remembered them to include impor¬ 
tant information that might be missing. 

Nevertheless, when specifically asked about the motivation promoted by the instruc¬ 
tional feedback technique, almost half of students indicated that they did not the tech¬ 
nique did not contribute to their motivation. This may have happened due to the differ¬ 
ences in student learning preferences, since the other half agreed that this technique has 
contributed to their motivation. Another reason may be lack of usage again due to the 
discreteness of the notifications, as these students also indicated that they would prefer 
to receive more intrusive feedback messages, ensuring they are going to read it in the 
appropriate moment in order to consequently take advantage of this technique. In this 
context, some students also suggested the use of some sound alert, informing when a 
new feedback message is delivered. Another improvement suggestion is to provide some 
functionality that could redirect to the screen to where the feedback suggestion may be 
applied. This is expected to assist the student to find where the correction may be done, 
even when the message is read long after its delivery. So, based on the collected informa¬ 
tion, we may infer that when students make proper use of this technique, it has a good 
chance to increase student motivation. 

9. Discussion 

Based on the perceptions of the students, it leads us to conclude that the instructional 
feedback technique is an important instrument to improve the learning of PM tools, and 
it may also increase their motivation during this process. 

The instructional feedback technique was designed to address content for each PM 
knowledge area, criticizing mistakes students typically make during the elaboration of 
the project charter and the project plan. The feedback messages have a satisfactory cov¬ 
erage of the addressed part of the PM process. Yet, it still can be improved perform¬ 
ing further analysis of the project charters and project plans delivered by students, and 
identifying recurrent mistakes, which would be avoided or minimize the delivery of new 
feedback messages. 

In relation to the instructional strategy, the immediate and non-intrusive delivery of 
the feedback has shown to be appropriated for the teaching of PM tools. However, we 
also identified several improvement opportunities. This includes a more obvious noti¬ 
fication when new messages are delivered as well as the flexibilization of the delivery 
strategy addressing these differences in the learning preference of the target audience. 
For instance, offering the possibility to configure either a less intrusive approach, ac¬ 
cumulating the feedback messages in the notification area, or a more intrusive approach, 
just displaying the feedback messages as soon as its trigger is fired. 

Through the implementation of the instructional feedback with respect to all PM 
knowledge areas in dotProject+, a professional web-based open-source PM tool can be 
used for teaching the usage of PM tools more effectively. As the feedback functionality 
is configurable, it can be turned on during the learning process and turned off during the 
professional usage of this tool. 
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And, although many students criticized some aspects of this technique, including the 
lack of depth or content coverage, almost 70% of the students agreed that this technique 
has positively contributed to their learning. So, the instructional feedback technique 
demonstrated that it may attend its purpose, while important improvement opportunities 
may be explored, aiming at addressing differences in learning preferences. 


9.1. Threats to Validity 

As any research, there are some threats to validity (Wohlin et al., 2012). Threats to 
conclusion validity may occur due to inconsistencies in the collected data. In this 
research the individuals may lack some knowledge related to PM, even while being 
taught about it during the course. It may lead to wrong interpretations of question¬ 
naire items and as a consequence lead to incorrect answers. To reduce this threat, the 
questionnaire was designed carefully analysing the wording, bringing it as near as 
possible to the students’ language. Another threat is related to the fact that the students 
answered the questionnaire during the course and may have been afraid to criticize 
some aspects of the proposed instructional feedback technique. This was mitigated by 
anonymising the answers and applying the questionnaire only after all student grading 
had been concluded. 

Threats to construction validity are related to the data collection instrument, which 
may not contain the necessary questions to reach the evaluation goal. To minimize this 
threat, we employed the GQM approach to design the evaluation, systematically de¬ 
composing the evaluation goal into analysis questions and metrics, which were then 
represented by questionnaire items. 

Threats to external validity may occur by not obtaining a significant sample. We 
mitigated that applying the IU in 4 classes, collecting data from a total of 51 students, 
from 3 different educational institutions. However, it is still important to conduct further 
studies, with the aim at generalizing the obtained results. 


10. Conclusion 

This article presents the design and evaluation of an instructional feedback technique, 
enhancing the teaching of PM tools aligned with PMBOK in higher computing educa¬ 
tion. It provides immediate and automatic feedback messages to students as they interact 
with the functionalities of a PM tool. Such messages include examples and suggestions 
about how to improve the project charter and the project plan developed using the PM 
tool. Its evaluation demonstrates the instructional feedback technique was welcomed by 
the students, providing a first indication that it can be a useful educational technique, 
being able to improve their learning and motivation during the instructional activities. 
Future studies could address several improvements suggested, such as different levels 
of feedback message intrusiveness and configuration of explanation depth, according to 
student learning preferences and performance. 
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